Automatic accuracy estimation for audio transcriptions

ABSTRACT

Embodiments of the present invention provide an approach for estimating the accuracy of a transcription of a voice recording. Specifically, in a typical embodiment, each word of a transcription of a voice recording is checked against a customer-specific dictionary and/or a common language dictionary. The number of words not found in either dictionary is determined. An accuracy number for the transcription is calculated from the number of said words not found and the total number of words in the transcription.

RELATED U.S. APPLICATION DATA

The present patent document is a continuation of U.S. patent applicationSer. No. 15/398,779, filed Jan. 5, 2017, the entire contents of whichare incorporated herein by reference, U.S. patent application Ser. No.15/398,779 is a continuation of U.S. patent application Ser. No.15/172,531, filed Jun. 3, 2016, U.S. Pat. No. 9,570,068, issued Feb. 14,2017, the entire contents of which are incorporated herein by reference,U.S. patent application Ser. No. 15/172,531 is a continuation of U.S.patent application Ser. No. 14/997,728, filed Jan. 18, 2016, U.S. Pat.No. 9,390,707, issued Jul. 12, 2016. U.S. patent application Ser. No.14/997,728 is a continuation of U.S. application Ser. No. 13/463,055,filed May 3, 2012, U.S. Pat. No. 9,275,636, issued Mar. 1, 2016, theentire contents of which are incorporated herein by reference.

TECHNICAL FIELD

In general, embodiments of the present invention provide an approach formultimedia processing. Specifically, embodiments of the presentinvention relate to an approach for automatically estimating theaccuracy level for audio transriptions.

BACKGROUND

Recent advances in media conversion technology, such as speech-to-textconversion and optical character recognition (OCR), have made itpossible for computers to perform tasks such as transcribing messages ordocuments dictated by a user. Dictation systems for personal computersare now fairly common. Also available are scanning or OCR systems thatare capable of converting the content of a printed document intocomputer readable form. These systems, however, are sometimesinaccurate. The users often need to proofread and make corrections tothe output of these systems.

Data entry personnel are currently utilized for the conversion of anaudio message to a text message sent to a user's paging device. In sucha system, a caller desiring to send a message telephones the user'spaging service and communicates the message to a data entry employee ofthe paging service. This employee enters the message into a computer andthen transmits it to the user's paging device. The text message enteredby the employee of the paging service is then displayed on the displayof the user's paging device. The use of human employees in the textentry and transcription of audio messages is expensive and inefficient.Current automated systems are not reliable and accurate enough to beused for a fully automated messaging or transcription system.

When documents are dictated and recorded as an audio file, a persontranscribing the document plays the audio file and enters textrepresenting the document recorded on the audio tape. The use of humanemployees in text entry and transcription of audio messages is expensiveand inefficient. Automated speech-to-text conversion systems are alsocurrently available to convert audio recordings into a text document.However, such conversion systems are inaccurate, requiring the users toproofread and make corrections to the output of the systems.

SUMMARY

In general, embodiments of the present invention provide an approach forestimating the accuracy of a transcription of a voice recording.Specifically, in a typical embodiment, each word of a transcription of avoice recording is checked against a customer-specific dictionary and/ora common language dictionary. The number of words not found in eitherdictionary is determined. An accuracy number for the transcription iscalculated from the number of said words not found and the total numberof words in the transcription.

A first aspect of the present invention provides a computer-implementedmethod for estimating the accuracy of a transcription of a voicerecording, comprising: receiving the transcription; providing a customerspecific dictionary; providing a dictionary of common language words;determining a number of inaccurate words in the transcription;determining a total number of words in the transcription; andcalculating an accuracy number based on the number of inaccurate wordsand the total number of words.

A second aspect of the present invention provides a system forestimating the accuracy of a transcription of a voice recording,comprising: a memory medium comprising instructions; a bus coupled tothe memory medium; and an audio transcription tool coupled to the busthat when executing the instructions causes the system to: receive thetranscription; provide a customer specific dictionary; provide adictionary of common language words; determine a number of inaccuratewords in the transcription; determine a total number of words in thetranscription; and calculate an accuracy number based on the number ofinaccurate words and the total number of words.

A third aspect of the present invention provides a computer programproduct for estimating the accuracy of a transcription of a voicerecording, the computer program product comprising a computer readablestorage medium, and program instructions stored on the computer readablestorage medium, to: receive the transcription; provide a customerspecific dictionary; provide a dictionary of common language words;determine a number of inaccurate words in the transcription; determine atotal number of words in the transcription; and calculate an accuracynumber based on the number of inaccurate words and the total number ofwords.

A fourth aspect of the present invention provides a method for deployinga system for identifying commands for estimating the accuracy of atranscription of a voice recording, comprising: providing a computerinfrastructure being operable to: receive the transcription; provide acustomer specific dictionary; provide a dictionary of common languagewords; determine a number of inaccurate words in the transcription;determine a total number of words in the transcription; and calculate anaccuracy number based on the number of inaccurate words and the totalnumber of words.

BRIEF DESCRIPTION OF THE DRAWINGS

These and other features of this invention will be more readilyunderstood from the following detailed description of the variousaspects of the invention taken in conjunction with the accompanyingdrawings in which:

FIG. 1 shows a computerized implementation of the present invention.

FIG. 2 shows a more detailed view of an example audio transcription toolaccording to an embodiment of the present invention.

FIG. 3 shows a method flow diagram for estimating the accuracy of atranscription of a voice recording according to an embodiment of thepresent invention.

FIG. 4 shows a method flow diagram for assigning a confidence level toinformation extracted from a transcript of a voice recording accordingto an embodiment of the present invention.

The drawings are not necessarily to scale. The drawings are merelyschematic representations, not intended to portray specific parametersof the invention. The drawings are intended to depict only typicalembodiments of the invention, and therefore should not be considered aslimiting the scope of the invention. In the drawings, like numberingrepresents like elements.

DETAILED DESCRIPTION

Embodiments of the present invention provide an approach for estimatingthe accuracy of a transcription of a voice recording. Specifically, in atypical embodiment, each word of a transcription of a voice recording ischecked against a customer-specific dictionary and/or a common languagedictionary. The number of words not found in either dictionary isdetermined. An accuracy number for the transcription is calculated fromthe number of said words not found and the total number of words in thetranscription.

FIG. 1 illustrates a computerized implementation 100 of the presentinvention. As depicted, implementation 100 includes computer system 104deployed within a computer infrastructure 102. This is intended todemonstrate, among other things, that the present invention could beimplemented within a network environment (e.g., the Internet, a widearea network (WAN), a local area network (LAN), a virtual privatenetwork (VPN), etc.), or on a stand-alone computer system. In the caseof the former, communication throughout the network can occur via anycombination of various types of communications links. For example, thecommunication links can comprise addressable connections that mayutilize any combination of wired and/or wireless transmission methods.Where communications occur via the Internet, connectivity could beprovided by conventional TCP/IP sockets-based protocol, and an Internetservice provider could be used to establish connectivity to theInternet, Still yet, computer infrastructure 102 is intended todemonstrate that some or all of the components of implementation 100could be deployed, managed, serviced, etc., by a service provider whooffers to implement, deploy, and/or perform the functions of the presentinvention for others.

Computer system 104 is intended to represent any type of computer systemthat may be implemented in deploying/realizing the teachings recitedherein. In this particular example, computer system 104 represents anillustrative system or constructing a SOA shared service. It should beunderstood that any other computers implemented under the presentinvention may have different components/software, but will performsimilar functions. As shown, computer system 104 includes a processingunit 106, memory 108 for storing an audio transcription tool 153, a bus110, and device interfaces 112.

Processing unit 106 collects and routes signals representing outputsfrom external devices 115 (e.g., a keyboard, a pointing device, adisplay, a graphical user interface, etc.) to audio transcription tool153. The signals can be transmitted over a LAN and/or a WAN (e.g., T1,T3, 56 kb, X.25), broadband connections (ISDN, Frame Relay, ATM),wireless links (802.11, Bluetooth, etc.), and so on. In someembodiments, the signals may be encrypted using, for example, trustedkey-pair encryption. Different external devices may transmit informationusing different communication pathways, such as Ethernet or wirelessnetworks, direct serial or parallel connections, USB, Firewire®,Bluetooth®, or other proprietary interfaces. (Firewire is a registeredtrademark of Apple Computer, Inc. Bluetooth is a registered trademark ofBluetooth Special Interest Group (SIG)).

In general, processing unit 106 executes computer program code, such asprogram code for operating audio transcription tool 153, which is storedin memory 108 and/or storage system 116. While executing computerprogram code, processing unit 106 can read and/or write data to/frommemory 108, common language dictionary 116, customer specific dictionary117, and a knowledge base (KB) 118. Common language dictionary 116,customer specific dictionary 117, and a knowledge base (KB) 118 caninclude VCRs, DVRs, RAID arrays, USB hard drives, optical diskrecorders, flash storage devices, or any other similar storage device.Although not shown, computer system 104 could also include I/Ointerfaces that communicate with one or more external devices 115 thatenable a user to interact with computer system 104.

FIG. 2 shows a more detailed view of an example audio transcription tool153. Audio transcription tool 153 includes an accuracy calculationcomponent 170 which calculates an accuracy of a transcription of a voicerecording. In addition, the audio transcription tool 153 includes aconfidence level assignment component 172 which assigns a confidencelevel to the transcription.

To determine a confidence level, a common language dictionary 116 and/ora customer specific dictionary 117 may be used. The common languagedictionary 116 may include a listing of correctly spelled words orcharacter strings for a particular natural language (e.g., English). Asused herein, the term “natural language” includes all punctuation,symbols, and numeric characters associated with a particular naturallanguage. In some examples, more than one common language dictionary maybe used.

The customer specific dictionary may include a listing of useful termsin the customer domain. For example, a customer in a hardware/power toolline may be provided one or more lists of products (obtained via adatabase). In some examples, more than one customer specific dictionarymay be used. In addition, one or more spell checkers may be used.

Knowledge base 118 may store various forms of knowledge. For example,knowledge base 118 may include automotive product information andcustomer shipping information. Knowledge base 118 is a collection ofknowledge stored in the form of “axioms”. An axiom is a statement orproposition that is regarded as being established, accepted, orself-evidently true. For example, storing genealogy information mayinclude the following axioms:

Father(Bill, Sara)

Father(John, Bill)

Grandfather(x, z):=Father(x, y), Father(y, z)

The axioms listed above relate that Bill is the father of Sara, John isthe father of Bill, and John is also the grandfather of Sara, Typicallyin building a KB (Knowledge Base), axioms are automatically extractedfrom data sources. Data sources are often free-form text. The process ofextraction requires NLP (Natural Language Processing) to interpret adata source and find potential axioms in the data source. Axioms thatare automatically discovered have a confidence level associated withthem. An axiom of lowest confidence might be rated a “1” and an axiom ofhighest confidence might be rated a “10”. Confidence levels may be usedbecause several potential answers may exist when a KB is used to answera question. The KB cannot determine which answer is correct, but canassign probabilities to the list (e.g., answer choice 1 has highestprobability of 63%, answer choice 2 is 20%, answer choice 1 is 17%).

Consider a customer support center that receives many customer supportcalls. Each of the calls are to be transcribed. The calls aretranscribed to text using software and recognized processes. However,the accuracy of the transcription may vary. Accuracy plays a large partin determining confidence levels. Confidence levels are relied upon bythe KB to return accurate responses to a user. If a transcript isinaccurate, an attempt is still made to extract axioms. A low confidencelevel may be associated with these extracted axioms. If a transcript isaccurate, a high confidence level may be associated with any extractedaxioms from the transcript.

FIG. 3 shows a method flow diagram for estimating the accuracy of atranscription of a voice recording according to an embodiment of thepresent invention. At step S1, a common language dictionary 116 isprovided. At step S2, a customer specific dictionary 117 is provided. Atstep S3, each word in the transcription of a voice recording are checkedto determine whether the word exists in the common language dictionary116 and/or the customer specific dictionary 117. In some examples, aspell check function is also performed on each word.

Any word that is not located in a dictionary and/or fails the spellcheck is determined to be an inaccurate word. For example, the word maybe misspelled. The greater the number of inaccurate words, the moreinaccurate the transcript. In step S4, an accuracy percentage may becalculated by simply dividing the number of accurate words by the totalnumber of words in the transcript. In one example, longer words may begiven more weight (e.g., words exceeding a predefined number ofcharacters). The weighting is based on the assumption that longer wordsare more relevant. A Gaussian function may then be used to find anexpected value and variance of each transcript per accuracy. Aconfidence level may be based on the output of the function.

FIG. 4 shows a method flow diagram for assigning a confidence level toinformation extracted from a transcript of a voice recording accordingto an embodiment of the present invention. In step S10, the transcripttext of a voice recording is received. In step S11, an axiom related tothe transcript text is extracted from a source. In step S12, aconfidence level of the source is determined. In step S13, a confidencelevel is assigned to the axiom based on the confidence level of thesource. In typical examples, more than one axiom is extracted related tothe transcript text.

Axioms are used in the computation when performing audio-to-texttranscribing to return results having confidence levels associated withthem. The use of axioms in the computation increases the likelihood thatat least one returned response to a question is an acceptable answer.

Below is an example of how the present invention may be used by anexample company (e.g., company ABC) having the following business case.Company ABC has millions of products and cannot compete on the value ofprice. Company ABC encourages customers to call in and get world-classcustomer support from customer service agents (CSAs) prepared to answerany question regarding any matter related to the company. The companyalso allows CSAs to ship any part to any destination within 24 hours atno cost to the customer.

Two levels of technical support exist at company ABC: L1 and L2. L1 mayhave new employees and trainees and is only prepared to take easiercalls. The L1 agents defer the more difficult calls to the L2 agents. L2contains the more veteran employees who are prepared to answer anyquestion. Company ABC wants to build a knowledge base (KB) that capturesknowledge and information about products and how they are used. The goalis to keep the L1 agents from deferring calls to the L2 agents byallowing the L1 agents to query the KB to answer any difficultquestions.

A customer calls the customer support line at Company ABC and says “Iwant a caulk for my kitchen”. An L1 agent is unsure how to answer anddecides to query the KB. The L1 agent manually types in “caulk forkitchen” or “I want a caulk for my kitchen”, or the like. The KB bringsback a response indicating three items that match the query. The L1agent describes the results (i.e., the three items) to the customer, thecustomer chooses one of the items (e.g., caulk XYZ) and the call isclosed.

There is no place within the the structured company ABC databases thatstates that caulk XYZ is “for use in kitchens”. If that data existed,then a search engine could return a result to the L1 agent. Rather, thestructured company ABC data states that caulk XYZ meets the regulation21 CFR 177.2600 specification (i.e., regulation number) and is used toget to the functional use (i.e., used in kitchens). CFR (Code of FederalRegulations) Title 21 is reserved for rules of the Food and DrugAdminstration. Unstructured data crawling is used to extract thefollowing:

-   -   21 CFR 2700.600 has Agency FDA    -   21 CFR 2700.600 has Topic Indirect Food Application    -   Indirect Food Application has Preferred Synonym Food Grade    -   Indirect Food Application has Environment Kitchen

Each statement in the preceding extracted data list is an axiom. Eachaxiom comes from a different source. For example, sources may includePDFs, web pages, web forums, speech-to-text transcripts, and the like.In other words, anything that contains ASCII text may be a sourcecandidate. A confidence level may be generated for each axiom becauseeach source cannot be be reviewed due the potential number of axioms andtime contraints.

A confidence level is automatically associated with each source whichcorresponds to an axiom when an axiom is extracted from speech-to-text(as described earlier). Confidence level has to do with the confidencein the source data that the axioms come from. Using the examples fromabove, associated confidence levels for each source is shown in Table 1.

TABLE 1 21 CFR 2700.600 has Agency FDA Confidence Level 5 21 CFR2700.600 has Agency Topic Confidence Level 4 Indirect Food ApplicationIndirect Food Application has Confidence Level 2 Preferred Synonym FoodGrade Indirect Food Application has Confidence Level 3 EnvironmentKitchen

An algorithm using axioms in the computation to return final results toa customer adds the confidence levels together. The answer with thehighest confidence level is most likely correct.

Further, it can be appreciated that the methodologies disclosed hereincan be used within a computer system to provide an audio transcriptionservice, as shown in FIG. 1. In this case, audio transcription tool 153can be provided, and one or more systems for performing the processesdescribed in the invention can be obtained and deployed to computerinfrastructure 102. To this extent, the deployment can comprise one ormore of (1) installing program code on a computing device, such as acomputer system, from a computer-readable medium; (2) adding one or morecomputing devices to the infrastructure; and (3) incorporating and/ormodifying one or more existing systems of the infrastructure to enablethe infrastructure to perform the process actions of the invention.

The exemplary computer system 104 may be described in the generalcontext of computer-executable instructions, such as program modules,being executed by a computer. Generally, program modules includeroutines, programs, people, components, logic, data structures, and soon that perform particular tasks or implements particular abstract datatypes. Exemplary computer system 104 may be practiced in distributedcomputing environments here tasks are performed by remote processingdevices that are linked through a communications network. In adistributed computing environment, program modules may be located inboth local and remote computer storage medium including memory storagedevices.

Furthermore, an implementation of exemplary computer system 104 may bestored on or transmitted across some form of computer readable media.Computer readable media can be any available media that can be accessedby a computer. By way of example, and not limitation, computer readablemedia may comprise “computer storage medium” and “communications media.”

“Computer storage medium” include volatile and non-volatile, removableand non-removable media implemented in any method or technology forstorage of information such as computer readable instructions, datastructures, program modules, or other data. Computer storage mediumincludes, but is not limited to, RAM, ROM, EEPROM, flash memory or othermemory technology, CD-ROM, digital versatile disks (DVD) or otheroptical storage, magnetic cassettes, magnetic tape, magnetic diskstorage or other magnetic storage devices, or any other medium which canbe used to store the desired information and which can be accessed by acomputer.

“Communication media” typically embodies computer readable instructions,data structures, program modules, or other data in a modulated datasignal, such as carrier wave or other transport mechanism. Communicationmedia also includes any information delivery media.

The term “modulated data signal” means a signal that has one or more ofits characteristics set or changed in such a manner as to encodeinformation in the signal. By way of example, and not limitation,communication media includes wired media such as a wired network ordirect-wired connection, and wireless media such as acoustic, RF,infrared, and other wireless media. Combinations of any of the above arealso included within the scope of computer readable media.

It is apparent that there has been provided with this invention anapproach for estimating the accuracy of a transcription of a voicerecording. While the invention has been particularly shown and describedin conjunction with a preferred embodiment thereof, it will beappreciated that variations and modifications will occur to thoseskilled in the art. Therefore, it is to be understood that the appendedclaims are intended to cover all such modifications and changes thatfall within the true spirit of the invention.

What is claimed is:
 1. A method of assigning a confidence level to atleast one axiom extracted from a text, comprising: determining a numberof accurate words in the text based on a spell check function; dividingthe number of accurate words from the text by a total number of words inthe text data; automatically extracting, using natural languageprocessing, from a knowledge base stored in a computer infrastructure,the at least one axiom, wherein the at least one axiom is associatedwith at least one word from the text, and wherein the axiom comprises acomputer-parsable definition of a relationship of data to at least oneof the words in the text; assigning a confidence level to the at leastone axiom based on a result of the dividing, wherein the confidencelevel is assigned based on an output of a Gaussian function applied tothe result of the dividing; receiving a query from a user device,wherein the query includes at least one word; matching the at least oneword with the at least one axiom using the knowledge base; and providingthe at least one axiom from the knowledge base to the user device basedon the matching and the assigned confidence level.
 2. The method ofclaim 1, further comprising: comparing the text to a dictionary.
 3. Themethod of claim 2, further comprising: determining a number ofinaccurate words in the text based on the comparing.
 4. The method ofclaim 2, wherein the dictionary comprises a customer specificdictionary.
 5. The method of claim 2, wherein the dictionary comprises adictionary of common language words.
 6. The method of claim 2, whereinthe text is extracted from multiple different sources.
 7. The method ofclaim 1, wherein the at least one axiom comprises a computer-parsabledefinition of a relationship of data to at least one word in atranscription of an audio recording.
 8. The method of claim 1, wherein asolution service provider provides a computer infrastructure operable toperform the method.
 9. A system for assigning a confidence level to atleast one axiom extracted from a text, comprising: a memory mediumcomprising instructions; a bus coupled to the memory medium; and anaudio transcription tool coupled to the bus that when executing theinstructions causes the system to: determine a number of accurate wordsin the text based on a spell check function; divide the number ofaccurate words from the text by a total number of words in the text;automatically extract, using natural language processing, from aknowledge base stored in a computer infrastructure, the at least oneaxiom, wherein the at least one axiom is associated with at least oneword from the text, and wherein the axiom comprises a computer-parsabledefinition of a relationship of data to at least one of the words in thetext; assign a confidence level to the at least one axiom based on aresult of the dividing, wherein the confidence level is assigned basedon an output of a Gaussian function applied to the result of thedividing; receive a query from a user device, wherein the query includesat least one word; match the at least one word with the at least oneaxiom using the knowledge base; and provide the at least one axiom fromthe knowledge base to the user device based on the matching and theassigned confidence level.
 10. The system of claim 9, the an audiotranscription tool coupled to the bus that when executing theinstructions further causes the system to: compare the text to adictionary.
 11. The system of claim 10, wherein the dictionary comprisesa customer specific dictionary.
 12. The system of claim 10, wherein thedictionary comprises a dictionary of common language words.
 13. Thesystem of claim 10, wherein the text is extracted from multipledifferent sources.
 14. The system of claim 9, wherein the at least oneaxiom comprises a computer-parsable definition of a relationship of datato at least one word in a transcription of an audio recording.
 15. Acomputer program product for assigning a confidence level to at leastone axiom extracted from a text, the computer program product comprisinga computer readable hardware storage device, and program instructionsstored on the computer readable hardware storage device, to: determine anumber of accurate words in the text based on a spell check function;divide the number of accurate words from the text by a total number ofwords in the text; automatically extract, using natural languageprocessing, from a knowledge base stored in a computer infrastructure,the at least one axiom, wherein the at least one axiom is associatedwith at least one word from the text, and wherein the axiom comprises acomputer-parsable definition of a relationship of data to at least oneof the words in the text; assign a confidence level to the at least oneaxiom based on a result of the dividing, wherein the confidence level isassigned based on an output of a Gaussian function applied to the resultof the dividing; receive a query from a user device, wherein the queryincludes at least one word; match the at least one word with the atleast one axiom using the knowledge base; and provide the at least oneaxiom from the knowledge base to the user device based on the matchingand the assigned confidence level.
 16. The computer program product ofclaim 15, the computer program product comprising a computer readablehardware storage device, and program instructions stored on the computerreadable storage device, to: compare the text to a dictionary.
 17. Thecomputer program product of claim 16, wherein the dictionary comprises acustomer specific dictionary.
 18. The computer program product of claim16, wherein the dictionary comprises a dictionary of common languagewords.
 19. The computer program product of claim 15, wherein the text isextracted from multiple different sources.
 20. The computer programproduct of claim 15, wherein the at least one axiom comprises acomputer-parsable definition of a relationship of data to at least oneword in a transcription of an audio recording.